Adaptive Streaming: A subjective catalog to assess the performance of objective QoE metrics

Scalable streaming has emerged as a feasible solution to resolve users' heterogeneity problems. Scalable Video Streaming (SVC) is the technology that has served as the definitive impulse for the growth of streaming adaptive systems. Systems need to incorporate user's Quality of Experience (QoE) in the layer switching algorithms. This paper characterizes the relation between human decisions and objective metrics that could be mapped into systems. We have performed extensive subjective experiments to corroborate the preference towards adaptive systems when compared to traditional non-adaptive systems. The resulting subjective scores are correlated with most relevant Full Reference (FR) objective metrics. We obtain an exponential relationship between human decisions and the same decisions expressed as a difference of objective metrics. A strong correlation with subjective scores validates objective metrics to be used as aid in the adaptive decision taking algorithms to improve overall systems performance. Results show that, among the evaluated objective metrics, PSNR is the metric that provide worse results in terms of reproducing the human decisions.